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ABSTRACT 

Eight language curricula that are widely used in the 
Netherlands for children up to grade 8 were evaluated. Questions 
examined wer«: (l) whether the use of a particular curriculum results 
in greater achievement in specific language areas; (2) what 
differences and similarities exist among these curricula; (3) do 
differences exist in the ways in which the curricula are implemented; 
and (4) in what settings is each used. Focus was on determining 
whether and to what degree the eight language curricula affected 
achievement in different language areas. In all, 110 schools and 
2,750 students in grade 8 participated in the evaluation. The data 
set is representative for the last grade of primary education in the 
Netherlands. The effects o* the curricula were assessed by a series 
of tests representing the curriculum's objectives. Although there 
were many practical and theoretical problems in comparing the 
curricula, analysis of variance did not indicate any differences in 
effectiveness among these curricula. It was not possible to calculate 
differences in the implementation of these curricula. These findings 
do not imply that education could do without language curricula; 
without them, good education is not possible. Teachers should be 
coached to implement these curricula in the proper way. When 
effective variables are built into the curriculum it might become 
more effective, and the determination of effectiveness might be 
enhanced. (SLD) 
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Language curricula in Dutch education 

Since the beginning of this century several language curricula 
have been brought onto the Dutch market for primary education. 
These curricula, which cover pupils 1 textbooks and subject matter 
and teacher guides (and additional resources) are developed for 
children in the ages six to twelve. The designers of language 
curricula are responsive to all kinds of educational reform. At 
the present moment primary schools can select from some fifteen 
different language curricula. For Dutch primary schools the 
choice of a new curriculum is a major event and takes place 
every five to ten years. Until now the choice of a language 
curriculum was made mainly on the basis of aspects of form. The 
realisation of specific educational objectives is not an 
important consideration choosing a curriculum (Harskamp et.- 
al. ,1982) . 

It is clear that language curricula play an important role in 
education and that the curricula have changed under the influence 
of time and educational reform. More recent curricula have 
facilitated differentiated instruction and the subject matter 
is more closely linked to pupils 1 domains of interest. The 
subject matter is more varied and is integrated by using a 
thematic approach. Besides the basic language skills all kinds 
of skills are taught. In the older curricula the accent was 

*) This article hat been made possible thanks to a subsidy 
granted by SV0- Institute for Educational Research in the 
Metherlands(SVO-project 4030) 
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mora on correct spelling and the proper use of grammar. The 
newer curricula, developed since the sixties, contain more 
productive language skills like speaking and creative writing. 
Another important innovation is a more individualized teaching 
approach. The new curricula offer sorts of content matter that 
make ability grouping possible. More recent curricula also offer 
tests that make an evaluation of pupils 1 progress easier and 
they are also better structured (Slavenburg, 1986) . The question 
arises whether educational innovations, as they have taken shape 
in language curricula, have different effects on pupils 1 langua- 
ge skills. For this reason eight different widely-used curricu- 
la were subjected to a critical evaluation. The following questi- 
ons were examined: 

1. Does the use of a certain language curriculum lead to greater 
achievements in specific language areas than the use of 
another? 

2. Which are the differences and similarities between language 
curricula? 

This question refers to the educational objectives which 
designers pursue in constructing their curriculum. 

3. Are there any differences in the way in which the curricula 
are implemented? 

4. What is the setting in which the curriculum is used?. 

This question refers to intervening pupil and teacher 
variables. 

For this research we selected eight curricula, covering about 
80% of the market. They do not only differ greatly with regard 
to the length of time they have been on the market, but also 
show large variations in the amount, the sequence and content 
of subject matter. For this reason it is not possible to create 
categories in which comparable language curricula are grouped 
together. 

The present study is aimed mainly at determining whether 
and to what degree the eight language curricula affected achieve- 
ment in different language areas (van Batenburg, 1988) . This 
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•valuation fits within the frame of reference of the so called 
'effective school 1 research tradition. This tradition forms a 
contrast to previous studies from which it was inferred that 
achievement is mainly affected by intelligence and social 
background. Given the central position language curricula have 
in Dutch education, it seems reasonable to attribute explanatory 
value not only to school and organizational variables but to 
language curriculum variables as well. 

Implementation and pupil and teacher characteristics 

To assume a direct relation between language curricula and achie- 
vement is not very realistic. A curriculum has to be implemented 
by the teacher, who does so in her own individual way, taking 
into account pupils 1 background and intelligence. The effect of 
language curricula can be assessed only if the implementation 
process and pupil and teacher characteristics are also taken into 
consideration. Of course there are numerous other variables that 
also affect achievement, such as family, peergroup and cultural 
variables. However, from the viewpoint of evaluation, it is 
advisable to use only a limited number of covariates. It is 
impossible to control for "everything". This consideration has 
lead to the following theoretical model of variables and 
influences: 
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Language curricula can influence achievement through the process 
of implementation. A curriculum that is not implemented can not 
have an effect. Pupils 9 and teacher characteristics influence 
achievement directly and indirectly via implementation. 

Design and data 

This investigation can be characterised as a program evaluation. 
We were inspired by the betting model (Hofstee, 1980) which is 
useful in solving evaluation problems . In this betting model 
agreement about effect criteria between designers of language 
curricula and researchers is essential. Agreement was sought 
primarily to avoid immunization strategies: designers of language 
curricula can then be 'pinned down 1 even when results are unfa- 
vourable for their curriculum. The designers were also asked to 
predict test scores of pupils who make use of their curriculum. 
This was done to assess the degree to which educational objecti- 
ves, as measured by the tests, were realized. 

The relative effectiveness of the curricula was measured by 
means of analysis of variance after controlling for relevant 
variables. These relevant covariables were found after partial 
correlation analysis and the fitting of a LISREL-model. This 
process was carried out in order to avoid overcorrection: we 
were only interested in the unique part of the variance in each 
variable. 

Before the data collection we had to take inventory of the 
use of the various curricula in schools. Our goal was to collect 
data from at least 15 schools in each curriculum. We did not 
achieve this number for three out of the eight language curricula 
because of a low degree of dissemination. In total 110 schools 
and 2750 grade 8 pupils cooperated in this investigation. A 
comparison of mean test scores with national norms showed no 
significant differences . This made us conclude that our data 
set is representative for the last grade of primary education 
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in Holland. In the last year of the primary school the chance 
is greatest that effects will have crystallised out. Pupils and 
teachers have used the curriculum then for six years. 

Instruments 

In the starting period of this investigation an analysis of the 
subject matter of several language curricula was carried out. 
This was done to obtain information about the similarities and 
differences between the "CITO-test" and the subject matter of 
the language curricula. The CITO-test (a standardized achieve- 
ment test for grade 8 pupils) covers mainly basic language skills 
like spelling, interpretative reading and grammar. For this 
reason, and to facilitate on the process of reaching agreement 
about effect criteria with designers, we constructed additional 
tests for speaking skills, (creative) writing skills, listeninr 
skills and critical reading skills. We succeeded in constructing 
reliable tests for all these skills except for speaking. For 
the measurement of speaking and creative writing skills judges 
were used, the other tests were of the multiple choice type. 
All these tests together cover most of the content of the subject 
matter in the language curricula. 

In a questionaire teachers were asked to what degree they 
individualise their instruction (differentiation) . This was done 
by means of an item scale (Reezigt, 1986) . Teachers were also asked 
how much time they spend on the different language skills. By 
means of two attitude scales it was assessed to what degree 
teachers operate in a manner which is 'pupil orientated' and 
•discipline orientated' (Meijnen 1980) . 

The social background of pupils was assessed in terms of the 
profession of the father in a variable with six categories from 
unskilled labourer up to academic professions. The assessment of 
intelligence was caried out by way of the non-verbal part of a 
general intelligence test(ISI, Boxtel, Snijders&Welten 1982). 
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Results 



Educational objectives 

When asked about the relative Importance of language tests the 
designers put more importance on the newly constructed tests. 
This result leads us to the conclusion that designers pay more 
attention to "modern ekills" than to the basic language skills. 
Their predictions of pupils test performance were inaccurate. 
Designers have little idea about the test scores of the pupils 
who ase their curriculum. For tests that assess basic skills 
they tend to under-estimate and for the other skills they tend 
to over-estimate the test performance of their pupils. The desig- 
ners place their priorities more on the modem educational 
objectives like creative writing, critical reading ability and 
listening, however without the expected results. 

Implementation 

The teachers 1 manuals for the language curricula were studied in 
order to find inscriptions for the way a curriculum should be 
used in the classroom by the teachers. Contrary to our expecta- 
tions in general no prescriptions were found. On the contrary 
teachers were advised to implement the curriculum in their own 
individual way. Because of this our theoretical distinction 
between "factual implementation" and "prescribed implementation" 
is useless, since there is no degree of implementation to which 
these two concepts could be compared. The concept of 
implementation has to do with the way the curriculum is used in 
the classroom. Indicators for this factual implementation are: 
the amount of subject matter which is handled, the amount of 
time spent on the diverse language skills and differentiation. 
About 80 percent of the subject matter has been taught at the 
end of the schoolyear. There are no differences here between 
the curricula. The proportion of time spent on the different 
language skills did not differ much between curricula. Nor did 
the way in which the teachers individualise their lessons. The 
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last point is quite striking because older methods are construc- 
ted only for frontal classroom teaching. These teachers tend to 
individualise anyhow. 

Teacher characteristics 

The differences between the teachers who work with the eight 
curricula on the two attitude scales 'pupil orientation' and 
'discipline orientation' proved to be non significant. The age 
of teachers of one curriculum was significantly lower (about 10 
years vith a mean age of 30) than the others. 

Pupil characteristics 

The differences between the curricula on the variable social 
background were significant. One curriculum, especially designed 
for children with disadvantaged background, was indeed in use 
with children from the lower economic stratum. The differences 
on the ncn-verbal intelligence test were not significant. 

Analysis of variance 

Before the analysis of variance could be performed it was nessa- 
sary to know which covariates had to be included in the model. 
The covariates were selected for their contribution of unique 
variance to achievement by way of partial correlation analysis. 
A potential covariate had to contribute unique variance after 
controlling for intelligence and background of the pupils. The 
remaining variables after this procedure were modelled in LISREL 
to trace indirect relations. The following model fitted the 
data well: 
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N*81; df=20 chi squared 5. 16 p=.20 

goodness of fit =.94 

* =significant path(T>2) 

The model shows that non-verbal intelligence is by far the best 
predictor of achievement. Social background and the two teacher 
attitude scales have a small but substantial contribution. The 
implementation variables showed no contribution in predicting 
achievement. 

The relative effect of language curricula was assessed by 
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Mans of analysis of variance on the level of the class. The 
design had three sources of variance in achievement: 

- the variance caused by non-verbal intelligence, social 
background, pupil orientation and discipline orientation; 

- variance due to the factor curriculum ( eight levels) ; 

- - residual variance ( variance cleared from the two above mentio- 

ned sources of variance) . 
Firstly the variance of the covariates was substracted and 
thereafter the variance of the curricula was tested against the 
residual variance. This was done multi- and univariate. 
The analysis of variance shoved no differences in effectiveness 
between curricula on all language tests. The curricula are all 
equally effective. 

Discussion 

The effect of language curricula was assessed by a series of 
language tests representing the curricula's objectives. In this 
evaluation agreement was reached upon the appropriateness of 
these language tests as measuring devices for the effects of 
their curricula. According to the designers the tests cover the 
subject matter sufficiently, and they can also be considered as 
relevant instruments for assessing effects. The problem of 
reaching agreement about the effect criteria was solved properly, 
immunisation strategies were avoided. The predictions made by 
the designers about the test scores of their pupils could offer 
a standard by which the degree to which their own educational 
objectives were realised could be assessed* However, because of 
the inaccuracy of these predictions, serious doubts could be 
raised about their value. In the betting model the quality of 
the predictions is linked to the quality of the curriculum, and 
because these predictions are made by the designers it might be 
possible that the quaxity of the predictions are also dependent 
on the intelligence of the designers. The intelligence or 
cleverness of designers is probably not the best indicator for 
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the quality of the curriculum. More interesting is the result 
that the implementation variables did not contribute to 
achievement. However without implementation of a curriculum ro 
effects can emerge. In that case one can speak of a non-event. 
There is often talk about a distinction between "factual 19 and 
"prescribed" or "desired" implementation. The degree of 
implementation or, implementation value, is the factual imple- 
mentation divided by the desired implementation . We were not 
able to calculate this implementation value because no 
prescriptions were made by designers in de teachers 1 manual how 
to implement the curriculum. We recommend that designers give 
more attention this problem and advise teachers more about 
proper use of the curriculum, so that a more effective 
implementation will be possible. The construction of good subject 
matter alone is not enough. 

The assessment of factual implementation is also problematic 
because the use of observations on a large scale is very expen- 
sive. Interesting instruction variables like tiiae per task could 
not be assessed in a valid way. The duration of language lessons 
and time spent on the different language skills are only meager 
indicators of the concept •time per task 1 . Other instructional 
variables could only be assessed in a rather flimsy manner (for 
example differentiation) , or not at all (feedback, re in f orcein an L) • 
There are indications that these variables have a greater influ- 
ence on achievement (see Fraser et al,1987). For our evaluation 
this would imply other covariates in the analysis of variance. 

The analysis of variance showed no differences in effective- 
ness between curricula. This finding does not imply that educati- 
on could do without language curricula. Without them, good educa- 
tion is not possible. However the pursuit of improvement by means 
of innovative curricula has not proved very realistic. We have 
seen that the teacher has a reasonable amount of influence. Tea- 
chers should therefore be coached to implement the method in 
the proper way. When effective variables are built into the 
curriculum it might be effective. Time per task, feedback, struc- 
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ture and reinforcement and other effective vari-bles could or 
should inspire the designers of the curricula. 
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